Lecture 12: Binocular Stereo and Belief Propagation

Binocular Stereo is the process of estimating three-dimensional shape (stereo) from two eyes (binocular) or two cameras. Usually it is just called stereo. It requires knowing the camera parameters (e.g., focal length, direction of gaze), as discussed in earlier lecture, and solving the correspondence problem – matching pixels between the left and right images. After correspondence has been solved then depth can be estimated by trigonometry (earlier lecture). The depth is inversely proportional to the disparity, which is the relative displacement between corresponding points. This lecture will discuss how to solve the correspondence to estimate the disparity can be formulated in terms of a probabilistic markov model which assumes that the surfaces of surfaces are piecewise smooth (like the weak membrane model). The epipolar line constraint (see earlier lecture) means that points in each image can only match to points on a one-dimensional line in the second image (provided the camera parameters are known). This enables a simplified formulation of stereo in terms of a series of independent one-dimensional problems which can each be formulated as inference on a graphical model without closed loops. Hence inference can be performed by dynamic programming (see earlier lecture). A limitation of these one-dimensional models is that they assume that the disparity/depth of neighboring pixels is independent unless they lie on the same epipolar line. This is a very restrictive assumption and it is better to impose dependence (e.g., smoothness) across the epipolar lines. But this prevents the use of dynamic programming. This motivates the use of the belief propagation (BP) algorithm which perform similarly to dynamic programming on models defined over graph structures without closed loops (i.e. are guaranteed to converge to the optimal estimate), but which also work well in practice as approximate inference algorithms on graphs with closed loops. The intuitive reasoning is that the one-dimensional models (which do dynamic programming) perform well on real images, hence introducing cross epipolar line terms will only have limited effects, and so belief propagation is likely to converge to the correct result (e.g., doing DP on the one-dim models which put you in the domain of attraction of the BP algorithm for the full model). This lecture will also introduce the belief propagation (BP) algorithm. In addition (not covered in class) we will describe the TRW and related algorithms. Note that since stereo is formulated as a Markov Model we can use a range of other algorithms to perform inference (such as the algorithms described earlier). Similarly we can apply BP to other Markov Models in vision. (Note that BP applies to Markov Models with pairwise connections but this can be extended to generalized belief propagation GBP which applies to Markov Models with higher order connections). Finally we point out a limitation of many current stereo algorithms. As known by Leonardo da Vinci, depth discontinuities in shapes can cause some points to be visible to one eye only, see figure (1). This is called half-occlusion. It affects the correspondence problem because it means that points in one eye may not have matching points in the other eye. Current stereo algorithms are formulated to be robust to this problem. However, the presence of unmatched points can be used as a cue to determine depth discontinuities and hence can yield information (this was done for one-dimensional models of stereo – Geiger, Ladendorf, and Yuille. Belhumeur and Mumford – which also introduced DP to this problem – except there was earlier work by Cooper and maybe Kanade?).